Display apparatus, remote controlling apparatus and control method thereof

ABSTRACT

A remote controlling apparatus, a display apparatus and a controlling method are provided. The remote controlling apparatus for selecting one of a plurality of operating modes of an external device being operable between a pointing mode and a gesture mode, associated with the remote controlling apparatus, includes an output unit for outputting information regarding the remote controlling apparatus to the external device, a detection unit for detecting motion of the remote controlling apparatus, a motion information generating unit for generating motion information based on the detected motion of the remote controlling apparatus, an operation mode change unit for providing information regarding an operation mode, for changing the operation mode of the external device being operable between the pointing mode and the gesture mode, and wherein the information regarding the remote controlling apparatus comprise the information regarding the operating mode, and the motion information generated by the motion information generating unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of application Ser. No.13/754,176, filed Jan. 30, 2013, which is now pending, and is relatedto, and claims priority to, Korean Patent Application No.10-2012-0066369 filed on Jun. 20, 2012 in the Korean IntellectualProperty Office, the entire disclosures of which are incorporated hereinby reference.

BACKGROUND

1. Field

Embodiments of the present disclosure relate to a remote controllingapparatus, a display apparatus, and control methods thereof. Moreparticularly, the embodiments of the present disclosure relate to aremote controlling apparatus for detecting a motion, a displayapparatus, and control methods thereof.

2. Description of the Related Art

As electronic technologies advance, methods for controlling anelectronic device in various manners are under development. Anelectronic device may be controlled by using a button of the electronicdevice or a remote control that is a separate device from the electronicdevice.

However, when the electronic device is controlled using a remote controlthat is separate from the electronic device, a user needs to identifyand press buttons of the remote control one-by-one for their intendedmanipulation.

For example, using a pointer displayed on a screen to select aparticular content on the electronic device screen, the user can selectthe particular content by moving the pointer to a corresponding contentregion by alternately selecting four direction buttons of the remotecontrol several times, and then pressing a selection button of theremote control in the corresponding content region. That is, the usercan select the corresponding content by identifying the buttons of theremote control multiple times and pressing the buttons multiple times.

Hence, what is needed is a method for a user to more easily discoverinformation displayed on the screen of the electronic device fromdistance.

SUMMARY

Additional aspects and/or advantages will be set forth in part in thedescription which follows and, in part, will be apparent from thedescription, or may be learned by practice of the invention.

The present invention addresses the above-mentioned and other problemsand disadvantages occurring in the conventional arrangement, amongothers. An aspect of the present invention provides a remote controllingapparatus for intuitively controlling a user interface screen (a UIscreen) provided from each mode of a display apparatus being operable ina plurality of operating modes, the display apparatus, and controlmethods thereof.

According to an aspect of the present invention, a remote controllingapparatus for providing remote control signals for an external displayapparatus having a plurality of operation modes includes a pointing modeand a gesture mode, and includes a signal output unit for outputting theremote control signals to the external display apparatus that provides auser interface screen, a detection unit for detecting motion of theremote controlling apparatus, an input unit comprising a first buttonunit and a second button unit, the first button unit being used forreceiving a user command for selecting one of the pointing mode and thegesture mode in the operation mode of the external display apparatus,and the second button unit being used for receiving a user command forselecting an pointing object displayed in the user interface screen, anda control unit for controlling the signal output unit to outputinformation regarding the detected motion and information regarding theuser command for selecting one of the pointing mode and the gesturemode, when the user command via the first button unit is input, and thepointing mode is a mode for controlling movement of the pointing objectdisplayed on the user interface screen according to the motion of theremote controlling apparatus detected by the detection unit, and thegesture mode is a mode for controlling display state of the userinterface screen according to a predefined gesture of the remotecontrolling apparatus.

During a time interval in which the information regarding the usercommand for selecting one of the pointing mode and the gesture mode isnot received from the remote controlling apparatus, the external displayapparatus may remain in the pointing mode as a default mode, and duringa time interval in which the information regarding the user command forselecting one of the pointing mode and the gesture mode is continuouslyreceived from the remote controlling apparatus, the external displayapparatus may transition to and remain in the gesture mode as anon-default mode.

The control unit may control the signal output unit to output theinformation regarding the user command for selecting one of the pointingmode and the gesture mode continuously while the first button unit isbeing pressed, and not to output the information regarding the usercommand for selecting one of the pointing mode and the gesture mode whenthe pressed first button unit is released

The detection unit may include at least one of an accelerometer, anangular velocity sensor, and a magnetometer.

A display apparatus having a plurality of operation modes comprising apointing mode and a gesture mode, controlled by a remote controllingapparatus that provides remote control signals, includes a receivingunit for receiving the remote control signals comprising informationregarding a motion of the remote controlling apparatus, informationregarding a user command for selecting one of the pointing mode and thegesture mode, and information regarding an item selection from theremote controlling apparatus, a display unit for displaying a userinterface screen having a pointing object which performs an indicatingfunction on the user interface screen, and a control unit forcontrolling to select one of the pointing mode and the gesture modebased on the information regarding the user command for selecting one ofthe pointing mode and the gesture mode received from the remotecontrolling apparatus, and the pointing mode is a mode for controllingmovement of the pointing object displayed on the user interface screenaccording to the motion of the remote controlling apparatus detected bythe detection unit, and the gesture mode is a mode for controllingdisplay state of the user interface screen according to a predefinedgesture of the remote controlling apparatus.

The display unit may display the pointing object of which a shape ischanged according to the operation mode.

The control unit may determine the operation mode of the displayapparatus as the pointing mode while the receiving unit does not receivethe information regarding the user command for selecting one of thepointing mode and the gesture mode, and the control unit may determinethe operation mode of the display apparatus as the gesture mode whilethe receiving unit receives the information regarding the user commandfor selecting one of the pointing mode and the gesture mode.

When the operation mode of the display apparatus is the pointing mode, amenu item included the user interface screen may be selected based onthe information regarding a motion of the remote controlling apparatusand the information regarding an item selection received from the remotecontrolling apparatus.

The control unit may control to display the pointing object by changingat least one of a shape, a color, a size, a location, and a directionaccording to an input manipulation state on the user interface screen.

The pointing object may be a hand-shaped pointer, and the control unitmay control a display of the hand-shaped pointer by changing into apointing type, a palm type, or a grip type based on a possiblemanipulation of flicking, zoom in/out, and panning, on the userinterface screen.

A display system comprising a remote controlling apparatus that providesremote control signals and a display apparatus having a plurality ofoperation modes comprising a pointing mode and a gesture mode,controlled by the remote controlling apparatus, includes the remotecontrolling apparatus for outputting the remote control signalscomprising information regarding a motion of the remote controllingapparatus, information regarding a user command for selecting one of thepointing mode and the gesture mode, and information regarding an itemselection, and the display apparatus includes a receiving unit forreceiving the remote control signals comprising information regarding amotion of the remote controlling apparatus, information regarding a usercommand for selecting one of the pointing mode and the gesture mode, andinformation regarding an item selection from the remote controllingapparatus, a display unit for displaying a user interface screen havinga pointing object which performs an indicating function on the userinterface screen, and a control unit for controlling to select one ofthe pointing mode and the gesture mode based on the informationregarding the user command for selecting one of the pointing mode andthe gesture mode received from the remote controlling apparatus, and thepointing mode is a mode for controlling movement of the pointing objectdisplayed on the user interface screen according to the motion of theremote controlling apparatus detected by the detection unit, and thegesture mode is a mode for controlling display state of the userinterface screen according to a predefined gesture of the remotecontrolling apparatus.

During a time interval in which information regarding the user commandfor selecting one of the pointing mode and the gesture mode is notreceived from the remote controlling apparatus, the display apparatusmay be configured to remain in a pointing mode as a default mode. Duringa time interval in which the information regarding the user command forselecting one of the pointing mode and the gesture mode is continuouslyreceived from the remote controlling apparatus, the display apparatusmay be configured to transition and remain in a gesture mode as anon-default mode.

A remote control apparatus may include a signal output unit foroutputting the remote control signals to the display apparatus, an inputunit having a first button for receiving the user command for selectingone of the pointing mode and the gesture mode, and a control unitconfigured to control the signal output unit to output the informationregarding the user command for selecting one of the pointing mode andthe gesture mode continuously while the first button unit is beingpressed, and not to output the information regarding the user commandfor selecting one of the pointing mode and the gesture mode when thepressed first button unit is released.

A control method of a remote controlling apparatus for providing remotecontrol signals for an external display apparatus having a pluralityoperation modes comprising a pointing mode and a gesture mode includesdetecting motion of the remote controlling apparatus and outputtinginformation regarding the detected motion and information regarding auser command for selecting one of the pointing mode and the gesturemode, when the user command for selecting one of the pointing mode andthe gesture mode in the operation mode of the external display apparatusis input, and the pointing mode is a mode for controlling movement of apointing object displayed on the user interface screen according to themotion of the remote controlling apparatus, and the gesture mode is amode for controlling display state of the user interface screenaccording to a predefined gesture of the remote controlling apparatus.

A control method of a display apparatus having a plurality of operationmodes comprising a pointing mode and a gesture mode, controlled by aremote controlling apparatus which provides remote control signalsincludes displaying a user interface screen having a pointing object forperforming an indicating function, receiving the remote control signalscomprising information regarding a motion of the remote controllingapparatus and information regarding a user command for selecting one ofthe pointing mode and the gesture mode from the remote controllingapparatus, and selecting one of the pointing mode and the gesture modebased on the information regarding the user command for selecting one ofthe pointing mode and the gesture mode received from the remotecontrolling apparatus.

In a pointing mode for controlling movement of a pointer displayed on auser interface screen provided by the display apparatus according tomotion of the remote controlling apparatus, if the user command isinput, the remote controlling apparatus may be switched to a gesturemode for controlling a display state of the user interface screenaccording to a predefined gesture of the remote controlling apparatus.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages of the present invention willbecome apparent and more readily appreciated from the followingdescription of the embodiments, taken in conjunction with theaccompanying drawings of which:

FIG. 1 illustrates a display system according to an embodiment of thepresent invention;

FIGS. 2A-2B illustrate exemplary operations of a display systemaccording to an embodiment of the present invention;

FIGS. 3A-3B illustrate a display apparatus according to an embodiment ofthe present invention;

FIG. 4 illustrates an exemplary control unit according to an embodimentof the present invention;

FIG. 5 illustrates software architecture of an exemplary storage unitfor supporting operations of a control unit according to an embodimentof the present invention;

FIG. 6 illustrates a remote controlling apparatus according to anembodiment of the present invention;

FIG. 7 illustrates the remote controlling apparatus ;

FIG. 8 illustrates a signal processing algorithm according to anembodiment of the present invention;

FIGS. 9A, 9B, and 9C illustrate an exemplary exterior of a remotecontrolling apparatus;

FIG. 10 illustrates a plurality of mode change button units of theremote controlling apparatus;

FIGS. 11A, 11B, and 11C illustrate manipulation of the remotecontrolling apparatus according to various embodiments of the presentinvention;

FIG. 12 illustrates operations in a pointing mode according to anembodiment of the present invention;

FIG. 13 illustrates operations in the mode change according to anembodiment of the present invention;

FIG. 14 illustrates operations in a gesture mode according to oneembodiment of the present invention;

FIG. 15 illustrates operations in the gesture mode according to anotherembodiment of the present invention;

FIG. 16 illustrates operations in the gesture mode according to yetanother embodiment of the present invention;

FIG. 17 illustrates a control method of a remote controlling apparatusaccording to an embodiment of the present invention; and

FIG. 18 illustrates a control method of the display apparatus accordingto an embodiment of the present invention.

DETAILED DESCRIPTION

Reference will now be made in detail to the embodiments of the presentinvention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The embodiments are described below to explain the presentinvention by referring to the figures.

FIG. 1 illustrates a display system according to an embodiment of thepresent invention.

As illustrated in FIG. 1, the display system according to an embodimentof the present invention includes a display apparatus 100 and a remotecontrolling apparatus 200.

The display apparatus 100 may be controlled by the remote controllingapparatus 200. While the display apparatus 100 can be implemented using,but not limited to, a digital TV, it can employ any device allowing theremote controlling such as a PC.

The remote controlling apparatus 200 remotely controls the displayapparatus 100. The remote controlling apparatus 200 can receive a usercommand and send a signal corresponding to the input user command to thedisplay apparatus 100.

The remote controlling apparatus 200 can detect a motion of the remotecontrolling apparatus 200 in a XYZ space and send a signal of thedetected 3D motion to the display apparatus 100. The 3D motion cancorrespond to a command for controlling the display apparatus 100. Thatis, a user can send a command to the display apparatus 100 by moving theremote controlling apparatus 200 in the space.

The remote controlling apparatus 200 may send motion information itselfcorresponding to the detected motion (a first exemplary embodiment), ormay convert motion information corresponding to the detected motion intoa control signal for controlling the display apparatus 100 and send thecontrol signal (a second exemplary embodiment). An operation forcontrolling the control signal from the detected motion information maybe performed by the display apparatus 100 or the remote controllingapparatus 200.

The display apparatus 100 can provide a plurality of different operatingmodes.

The display apparatus 100 can operate in a pointing mode for controllingmovement of a pointing object displayed in a user interface screen(hereafter, referred to as a UI screen) provided through the displayapparatus 100 according to the motion of the remote controllingapparatus 200, and in a gesture mode for controlling the display stateof the UI screen according to a predefined gesture of the remotecontrolling apparatus 200. The change of the display state of the UIscreen controlled in the gesture mode may change the screen displaystate according to a preset event, such as screen change executing aparticular function such as menu screen display or a screen change suchas page change for changing contents displayed in the screen. Thegesture mode can be differentiated from the pointing mode which merelycontrols the movement of the pointing object.

The pointing mode may be provided based on an absolute coordinate schemein which motion region of the remote controlling apparatus 200 and thescreen of the display apparatus 100 are matched.

The motion region of the remote controlling apparatus 200 correspondingto the whole screen region of the display apparatus 100 can be preset.For example, the motion region between left-handed rotation 30° andright-handed rotation 30° in a preset reference state can correspond tothe whole display screen. A preset reference state can be set by a usercommand. For example, when the remote controlling apparatus 200 isturned on and the user command for setting the reference state (e.g., acentering button (not illustrated) of the remote controlling apparatus200) is input, the current state of the remote controlling apparatus 200can be set to the reference state and the pointing object correspondingto the current state of the remote controlling apparatus 200 can bedisplayed at a preset position (e.g., in the center of the screen) inthe preset region of the screen of the display apparatus 100.

A coordinate value of the pointing object corresponding to the rotationstate of the remote controlling apparatus 200 moved by the user can becalculated based on a preset reference pint. For example, X and Ycoordinate values corresponding to the rotation state of the remotecontrolling apparatus 200 moved by the user can be calculated using thecenter of the display screen as the absolute reference point.

For example, when the remote controlling apparatus 200 rotates 10degrees to the right in the space, the pointing object corresponding tothe rotation state of the remote controlling apparatus 200 can move fromthe matching point (x1, y1) (e.g., the center pointer) to (x2, y2).

A relative coordinate scheme may be applied, which is well known in theart and its detailed description is omitted.

The remote controlling apparatus 200 can include a mode change buttonfor receiving the user command to change the operation mode of thedisplay apparatus 100. When the mode change button is being pressed, thechanged mode can be maintained.

The display apparatus 100 and the remote controlling apparatus 200 cancommunicate using various communication schemes such as BlueTooth (BT),Zigbee, Wireless Fidelity (Wi-Fi), Infrared (IR), Serial Interface, andUniversal Serial Bus (USB). For example, when the display apparatus 100and the remote controlling apparatus 200 communicate using the BT, theycan interwork through BT pairing. A BT pairing is well-known to thoseskilled in the art and its detailed explanations is omitted.

Operations of an exemplary embodiment are disclosed.

A First Exemplary Embodiment

FIG. 2A illustrates of operations of a display system according to thefirst exemplary embodiment of the present invention.

As illustrated in FIG. 2A, a mode change command may be received fromthe remote controlling apparatus 200 (S11). For example, if the displayapparatus 100 operates in the pointing mode, the display apparatus 100may receive a mode change command to change the pointing mode into thegesture mode, or if the display apparatus 100 operates in the gesturemode, the display apparatus 100 may receive a mode change command tochange the gesture mode into the pointing mode. The display apparatus100 can provide the UI screen including the pointing object. In thepointing mode, the pointing object displayed in the UI screen can pointat least one content, and the pointer can move on the UI screenaccording to the rotation direction and the rotation degree of theremote controlling apparatus 200. For example, when the pointing objectis in a hand shape, a hand-shaped pointing type pointer can be used.When the absolute coordinate scheme is used in the pointing mode, thedisplay position of the pointing object displayed in the screen can bemoved.

In the gesture mode, the display apparatus 100 can change and displaythe hand-shaped pointing type pointer to a palm-type hand pointer.

The display apparatus 100 can display a guide GUI for guiding thegesture input together with the palm-type hand pointer on the screen.The guide GUI can be in a different shape according to the type of theUI screen. For example, the guide GUI can use an arrow for guiding thegesture direction applicable to the current UI screen.

If the remote controlling apparatus 200 detects a motion (S12), theremote controlling apparatus 200 generates motion informationcorresponding to the detected motion (S13). For example, the remotecontrolling apparatus 200 may generate motion information correspondingto “rotation of a 5° angle to the right”.

The remote controlling apparatus 200 transmits to the display apparatus100 information regarding the operation mode of the display apparatus100 according to the mode change command received in operation of S11and the motion information generated in operation of S13 (S14). Theinformation regarding the operation mode may be information indicatingthe pointing mode or the gesture mode.

The display apparatus 100 generates a control signal for controllingoperation of the display apparatus 100 based on the received informationregarding the operation mode and the received motion information (S15).

if the received mode information indicates the pointing mode, thereceived motion information may be converted and generated into acontrol signal corresponding to the pointing mode, or if the receivedmode information indicates the gesture mode, the received motioninformation may be converted and generated into a control signalcorresponding to the gesture mode. For example, if the receivedinformation regarding the operation mode indicates the pointing mode andif the received motion information indicates “rotation of a 10° angle tothe right”, a control signal may be generated to move the pointingobject displayed on the screen to corresponding coordinates. if thereceived mode information indicates the gesture mode, if the receivedmotion information indicates “rotation of a 10° angle to the right”, andif a gesture corresponding to the motion information is mapped to ascreen change command, a control signal may be generated to change a UIscreen page displayed on the screen to a subsequent UI screen page.

The display apparatus 100 performs operation corresponding to thegenerated control signal (S16).

FIG. 2B illustrates exemplary operations of a display system ofaccording to a second exemplary embodiment of the present invention.

Operations of S21-S23 illustrated in FIG. 2B are similar tocorresponding operations of FIG. 2A, so detailed description is notrepeated.

The remote controlling apparatus 200 generates a control signal based onthe mode information included in the mode change command received inoperation of S21 and the motion information generated in operation ofS23 (S24). That is, if the received mode information indicates thepointing mode, the received motion information may be converted andgenerated into a control signal corresponding to the pointing mode, orif the received mode information indicates the gesture mode, thereceived motion information may be converted and generated into acontrol signal corresponding to the gesture mode. Accordingly, theremote controlling apparatus 200 may be implemented with an arithmeticoperation function for generating a control signal based on the receivedmode information and motion information.

The remote controlling apparatus 200 transmits the generated controlsignal to the display apparatus 100 (S25).

The display apparatus 100 performs operation corresponding to thereceived control signal (S26). That is, the display apparatus 100 mayperform its operation without an arithmetic operation unlike theexemplary embodiment illustrated in FIG. 2A.

For example, if the display apparatus 100 receives a control signalcorresponding to a motion which changes a currently displayed UI page toa subsequent UI page, the display apparatus 100 performs the operationcorresponding to the received control signal.

FIGS. 3A-3B illustrate a display apparatus according to an embodiment ofthe present invention.

Referring to FIG. 3A, the display apparatus 100 may include a receiver110, a display unit 120, and a control unit 130. An exemplary operationof the display apparatus 100 is disclosed according to a first andsecond exemplary embodiments.

An Operation of the Display Apparatus According to the First ExemplaryEmbodiment

While the display apparatus 100 can be implemented using, but notlimited to, a digital TV, a PC, or a notebook, it can employ any devicehaving the display function and allowing the remote controlling.

The receiver 110 can communicate with the remote controlling apparatus200. The receiver 110 can communicate with the remote controllingapparatus 200 using the various communication schemes such as BT,Zigbee, Wi-Fi, IR, Serial Interface, and USB.

The communication unit 110 can receive information regarding anoperation mode of mode of the display apparatus 100 and a remote controlsignal including motion information of the remote controlling apparatus200 from the remote controlling apparatus 200. The information regardingthe operation mode of the display apparatus 100 may be informationregarding a user command to select one of a pointing mode or a gesturemode, and may be implemented in diverse forms. For example, the modeinformation may be implemented in a flag form. In the case of thepointing mode, the flag may indicate “0”, or in the case of the gesturemode, the flag may indicate “1”. That is, the remote controllingapparatus 200 may generate flag information indicating “0” or “1”according to the user's command to change the mode. The user command formode selection may be input through a first button unit (notillustrated) formed on the remote controlling apparatus 200.

The information regarding the operation mode may not only be transmittedto the display apparatus 100 whenever the motion information istransmitted, but may also be transmitted only when there is a modechange command. In the latter case, the display apparatus 100 recognizesmotion information that is received until information regarding achanged operation mode is retransmitted after information regarding theoperation mode is transmitted, as motion information for the sameoperating mode.

The receiver 110 may receive information regarding item selection fromthe remote controlling apparatus 200. For example, the receiver 110 mayreceive a selection signal to select an item where the pointing objectis located in the pointing mode, and the selection signal may be inputthrough a second button unit (not illustrated) of the remote controllingapparatus 200.

The receiver 110 may be configured to perform a transmission function totransmit information to the remote controlling apparatus 200. Forexample, when the display apparatus 100 is turned off, the remotecontrolling apparatus 200 can be automatically turned off by sending apower-off signal to the remote controlling apparatus 200.

The display unit 120 can provide various display screens through thedisplay apparatus 100.

The display unit 120 can display a UI screen having a pointing objectwhich performs an indicating function.

The display unit 120 can display the GUI corresponding to the operationmode of the display apparatus 100 under control of the control unit 130.

In the pointing mode, the display unit 120 can display the pointingobject such as cursor, mouse cursor, or highlight display on thedisplayed UI screen. In the pointing mode, the display unit 120 can moveand display the position of the pointing object according to motioninformation received from the remote controlling apparatus 200.

The display unit 120 may display the pointing object of which shape ischanged according to operation modes.

When the display apparatus 100 is switched from the pointing mode to thegesture mode, the display unit 120 can display the guide GUI for guidingthe gesture input on the displayed UI screen. The guide GUI can be, butnot limited to, an arrow-shaped cue indicating the gesture direction.

The display unit 120 can be implemented using various displays such asliquid crystal display, thin film transistor-liquid crystal display,organic light-emitting diode, flexible display, and 3D display.

The control unit 130 controls the operations of the display apparatus100.

The control unit 130 can control selection of an operation mode based onthe information regarding the operation mode received from the remotecontrolling apparatus 200.

The control unit 130 can control selection of one of a pointing mode anda gesture mode based on information regarding a user command to selectone of the pointing mode and the gesture mode.

A control unit 130 can determine the operation mode of the displayapparatus 100 as the pointing mode while information regarding a usercommand to select one of the pointing mode and the gesture mode is notreceived, and can determine the operation mode of the display apparatus100 as the gesture mode while information regarding a user command toselect one of the pointing mode and the gesture mode is received.

The controller 130 can convert motion information received from theremote controlling apparatus 200 into a control signal to control thedisplay apparatus 100 and control the operation of the display apparatus100 according to the converted control signal.

The control unit 130 can convert the received motion information into acontrol signal applied to the function of the display apparatus 100using a control signal mapping table, and control the functions of thedisplay apparatus 100 using the converted control signal.

The control unit 130 may control the display unit 120 to provide the GUIof the corresponding type according to the operation mode of the displayapparatus 100.

For example, when the UI screen is displayed in the display unit 120,the display apparatus 100 can operate in the pointing mode and displaythe corresponding GUI. When the display apparatus 100 operates in thegesture mode according to the received information, the display unit 120can be controlled to display the corresponding GUI. The GUI is apointing object for various indicating functions on the UI screen andcan employ the pointer, the cursor, or the mouse cursor as mentionedabove.

If the operation mode of the display apparatus 100 is the pointing mode,a menu item included in the UI screen can be selected based oninformation regarding the motion of the remote controlling apparatus 200and information regarding item selection received from the remotecontrolling apparatus 100.

When the display apparatus 100 receives mode information correspondingto the gesture mode from the remote controlling apparatus 200, thecontrol unit 130 can display the type of the pointing object displayedon the screen. For example, when the pointing object is the hand-shapedpointer, the hand pointing shape with only the forefinger spread can bedisplayed in the pointing mode and the palm-type hand with all fivefingers spread can be displayed in the gesture mode.

The control unit 130 can change and display at least one of the shape,the color, the size, the position, and the direction of the pointingobject according to the input manipulation on the UI screen displayed inthe gesture mode. For example, when the pointing object is thehand-shaped pointer and the screen zoom in/out operation is available inthe gesture mode, the hand-shaped pointer can be enlarged in the zoom-instate and reduced in the zoom-out state.

In the gesture mode, the control unit 130 can control the display unit120 to display a guide GUI having the direction guiding the gestureinput through the remote controlling apparatus 200. For example, whenleft and right flicking operations are possible in the gesture mode, theguide GUI of the left and right arrow directions can be displayed.

The control unit 130 can control the display unit 120 to change anddisplay at least one of the movement direction and the movement degreeof the pointing object displayed on the UI screen according to at leastone of the rotation direction and the rotation degree of the remotecontrolling apparatus 200 in the pointing mode. An absolute coordinatescheme can be used.

When the gesture for rotating the remote controlling apparatus 200 onthe XY axes in parallel with the screen of the display unit 120 is inputin the gesture mode, the control unit 130 can control the display unit120 to change and display the UI screen to other UI screensystematically matched to the XY-axis directions.

When the gesture for rotating the remote controlling apparatus 200 onthe Z axis which is vertical to the screen of the display unit 120 isinput in the gesture mode, the control unit 130 can control to zoom inor out the selected content on the UI screen according to the rotationof the remote controlling apparatus 200.

An Exemplary Operation of the Display Apparatus According to a SecondExemplary Embodiment

In the operation of the display apparatus according to a secondexemplary embodiment, detailed description of components that aresimilar to those of the display apparatus according to the firstexemplary embodiment are not repeated.

The communication unit 110 may communicate with the remote controllingapparatus 200. The communication unit 110 may communicate with theremote controlling apparatus 200 using the various communication schemessuch as BT, Zigbee, Wi-Fi, IR, Serial Interface, and USB.

The communication unit 110 may receive a control signal generated basedon mode information input by the remote controlling apparatus 200 andmotion information of the remote controlling apparatus 200. That is,unlike the first exemplary embodiment, the receiver 110 may receive fromthe remote controlling apparatus 200 a control signal of a form capableof controlling the display apparatus 100 without a separate arithmeticoperation.

The display unit 120 may provide diverse display screens that can beprovided by the display apparatus 100. Since content displayed by thedisplay unit 120 is similar as that of the first exemplary embodiment,detailed description is not repeated.

The control unit 130 may control the operation of the display apparatus100 based on a control signal received from the remote controllingapparatus 200. The control unit 130 may control the movement of thepointing object according to a received control signal in the pointingmode, and may change screen according to a received control signal inthe gesture mode. Since the control of the display unit 120 by thecontrol unit 130 has been described in the first exemplary embodiment,detailed description is not repeated.

FIG. 3B illustrates of an exemplary display apparatus.

Components of the display apparatus according to a first exemplaryembodiment are explained for convenience of description. Since thedisplay apparatus according to a second exemplary embodiment operatesaccording to a control signal transmitted from the remote controllingapparatus, detailed description is not repeated.

Referring to FIG. 3B, a display apparatus 1000 includes thecommunication unit 110, the display unit 120, the control unit 130, astorage unit 140, an image receiving unit 150, an image processing unit160, a user interface unit 170, and a UI processing unit 180. In FIG.3B, the display apparatus 100 of FIG. 3A may be implemented using thedigital TV. A description of components in FIG. 3B that are similar tothe components of FIG. 3A are omitted.

The storage unit 140 is a storage medium storing various data andprograms for operating the display apparatus 1000, and can beimplemented using a memory or a Hard Disk Drive (HDD).

The storage unit 140 can store the control information corresponding tothe motion information received from the remote controlling apparatus200 in the gesture mode. The storage unit 140 can store the controlsignal corresponding to the motion information received from the remotecontrolling apparatus 200 in the gesture mode, in a table form. Forexample, a control signal corresponding to the UI screen change commandcan be matched and stored when the motion information corresponding tothe gesture (e.g., the left and right rotation of the remote controllingapparatus 200) flicked from the left to the right is received from theremote controlling apparatus 200. A control signal corresponding to thecontent zoom in/out command can be matched and stored when the motioninformation corresponding to the forward and backward movement gesture(e.g., the forward and backward rotation of the remote controllingapparatus 200) is received from the remote controlling apparatus 200.

The image receiving unit 150 receives the broadcasting content over anantenna or a cable, or receives the image content from an externaldevice or an external communication network.

The image receiving unit 150 can receive various image contents over anetwork or over the air. The contents can be various contents such asthe produced contents such as VOD content or broadcasting contents.

The image receiving unit 150 can be received in various types. Forexample, when the display apparatus 100 is implemented using the digitalTV, the image receiving unit 150 can be implemented using a set-top box,a tuner, an external input port, and/or a network communication module.

The image processing unit 160 processes various signals received throughthe image receiving unit 150. An image processing unit 160 can includesignal processing components such as demodulator, decoder, A/Dconverter, and scaler.

The user interface unit 170 receives various user signals.

The user interface unit 170 can be implemented in various fashionsaccording to the type of the display apparatus 100. For example, whenthe display apparatus 100 is implemented using the digital TV, the userinterface unit 170 may be realized as the communication unit 110 forreceiving the remote control signal.

The user interface unit 170 may be implemented using an input panelincluding a plurality of keys, or a touch screen of a layer structureincluding a display and a touch pad.

The UI processing unit 180 generates various UI elements superimposed onthe image output on the display unit 120 under the control of thecontrol unit 130. The UI processing unit (not illustrated) can generate2D or 3D UI elements.

The UI processing unit 180 can process 2D/3D conversion, transparency,color, size, shape and location adjustment, highlight, and animationeffect of the UI element under the control of the control unit 130.

FIG. 4 illustrates a control unit 130 according to an embodiment of thepresent invention.

Referring to FIG. 4, the control unit 130 includes a system memory 131,a main CPU 132, an image processor 133, a network interface 134, astorage interface 135, first through n-th interfaces 136-1 through136-n, an audio processor 137, and a system bus 138.

The system memory 131, the main CPU 132, the image processor 133, thenetwork interface 134, the storage interface 135, the first through n-thinterfaces 136-1 through 136-n, and the audio processor 137 areinterconnected through the system bus 138 to transmit and receivevarious data and signals.

The first through n-th interfaces 136-1 through 136-n support theinterfacing between the various components including the display unit120 and the components of the control unit 130. At least one of thefirst through n-th interfaces 136-1 through 136-n may be implementedusing a button of the main body of the display apparatus 100, or aninput interface for receiving various signals from an external deviceconnected via first through n-th external input ports.

The system memory 151 includes a ROM 131-1 and a RAM 131-2. The ROM131-1 stores an instruction set for booting up the system. When aturn-on instruction is input and the power is supplied, the main CPU 132copies Operating System (O/S) stored to the storage unit 140 to the RAM131-2 and boots up the system by executing the O/S according to theinstruction stored to the ROM 131-1. When the booting is completed, themain CPU 132 copies various application programs stored to the storageunit 140 to the RAM 131-2, and executes various operations by runningthe application programs copied to the RAM 131-2.

The main CPU 132 can perform various operations according to theexecution of the application programs stored to the storage unit 140.

The storage interface 135 may be connected to the storage unit 140, andtransmits and receives various programs, contents, and data.

For example, when the control signal is received from the remotecontrolling apparatus 200, the main CPU 132 can read the controlinformation mapped to the control signal by accessing the storage unit140 via the storage interface 135, and then control the displayapparatus 100 using the read control information.

The image processor 133 can include a decoder, a renderer, a scaler, forexample. The image processor 133 decodes the stored content, constructsa frame by rendering the decoded content data, and scales the size ofthe constructed frame in accordance with the screen size of the displayunit 120. The image processor 133 provides the processed frame to thedisplay unit 120 to display the frame.

The audio processor 137 processes and provides audio data to a devicethat produces a sound output such as speaker (not illustrated). Theaudio processor 137 can decode the audio data stored to the storage unit140 or the audio data received from the outside, filter noise, and thenprocess the audio signal, for example, amplify the audio to an adequatedecibel. For example, when the played content is a video content, theaudio processor 137 can process audio data demultiplexed from the videocontent and provide the processed audio data to the speaker (notillustrated) to output the audio in synchronization with the imageprocessor 133.

The network interface 134 may be connected to external devices over anetwork. For example, when the application for providing informationprovision service is driven, the main CPU 132 can communicate with theremote controlling apparatus 200 via the network interface 134.

Operations of the control unit 130 can be executed by running variousprograms stored to the storage unit 140.

FIG. 5 illustrates exemplary software architecture of the storage unit140 for supporting the operations of the control unit 130 according tovarious embodiments of the present invention. Referring to FIG. 5, thestorage unit 140 includes a base module 510, a device management module520, a communication module 530, a presentation module 540, a webbrowser module 550, and a service module 560.

The base module 510 is a base module for processing signals receivedfrom the hardware of the display apparatus 100 and sending the processedsignals to a higher layer module.

The base module 510 includes a storage module 511, a location basedmodule 512, a security module 513, and a network module 514.

The storage module 511 is a program module for managing database (DB) orregistry The location based module 512 is a program module forsupporting a location based service in association with hardware such asGPS chip. The security module 513 is a program module for supportinghardware certification, permission, and secure storage. The networkmodule 514 supports the network connection and includes a DNET moduleand an UPnP module. For example, the network module 514 can access anexternal server (not illustrated) over the network.

The device management module 520 manages and utilizes information of theexternal input and the external device. The device management module 520can include a sensing module 521, a device information management module522, and a remote control module 523. For example, information of theremote controlling apparatus 200 can be managed through the devicemanagement module 520.

The sensing module 521 analyzes various sensor data. For example, thesensing module 521 can include a face recognition module, a voicerecognition module, a motion recognition module, and an NFC recognitionmodule.

The device information management module 522 provides information aboutvarious devices, and the remote control module 523 is a program modulefor remotely controlling peripheral devices such as phone, printer,camera, and air conditioner.

The communication module 530 is a module for communicating with theoutside. The communication module 530 can include a messaging module 531such as messenger program, Short Message Service (SMS) and MultimediaMessage Service (MMS) program, and Email program, and a telephony module532 including a call info aggregator program module and VoIP module.

The presentation module 540 is a module for generating the displayscreen. The presentation module 540 includes a multimedia module 541 forplaying and outputting multimedia content, and a UI & graphics module542 for processing the UI and graphics. The multimedia module 541 caninclude a player module, a camcorder module, and a sound processingmodule. A multimedia module 541 plays various multimedia contents, andgenerates and plays the screen and the sound. The UI & graphics module542 can include an image compositor module 542-1 for combining images, acoordinate combination module 542-2 for combining and generatingcoordinates on the screen for displaying the image, an X11 module 542-3for receiving events from the hardware, and a 2D/3D UI toolkit 542-4 forproviding a tool for creating the 2D or 3D UI. For example, the UIscreen corresponding to the pointing mode or the gesture mode can begenerated through the presentation module 540.

The web browser module 550 accesses a web server through web browsing.The web browser module 550 can include various modules such as web viewmodule for creating the web page, a download agent module fordownloading, a bookmark module, and webkit module.

The service module 560 indicates an application module for providingvarious services. For example, the service module 560 can includevarious modules such as navigation service module for providing map,current location, landmark, and route information, game module, andadvertisement application module.

The main CPU 132 of the control unit 130 accesses the storage unit 140through the storage interface 135, duplicates the various modules storedto the storage unit 140 to the RAM 131-2, and operates according to theoperation of the duplicated module.

For example, when the GUI display corresponds to the pointing mode orthe gesture mode, the main CPU 132 generates the GUI screen using theimage combination module 542-1 of the presentation module 540. The mainCPU 132 determines the display location of the GUI screen using thecoordinate combination module 542-2 and controls the display unit 120 todisplay the GUI screen at the location.

When the user manipulation corresponds to the message reception, themain CPU 132 accesses a message management server and then receives amessage stored to a user account by executing the messaging module 541.The main CPU 132 generates a screen corresponding to the receivedmessage using the presentation module 540 and displays the screen on thedisplay unit 120.

As for a phone call operation, the main CPU 132 may drive the telephonymodule 532.

The storage unit 140 can store programs of diverse structures, and thecontrol unit 130 can operate using the various programs stored to thestorage unit 140 according to various embodiments of the presentinvention.

FIG. 6 illustrates a remote controlling apparatus according to anembodiment of the present invention.

An Exemplary Operation of a Remote Controlling Apparatus According tothe First Exemplary Embodiment

Referring to FIG. 6, the remote controlling apparatus 200 includes adetection unit 210, an input unit 220, a signal output unit 230, and acontrol unit 240.

The remote controlling apparatus 200 provides a remote controllingsignal regarding the external display apparatus 100 which has aplurality of operation modes including the pointing mode and the gesturemode.

The detection unit 210 detects 3D motion of the remote controllingapparatus 200. The detection unit 210 can include at least one of anaccelerometer, a gyro sensor, and a magnetometer. A combination of oneor two or more sensors of the detection unit 210 can detect the 3Dmotion of the remote controlling apparatus 200. The 3D motion can be therotation of the remote controlling apparatus 200.

The accelerometer measures spatial motion of the remote controllingapparatus 200. That is, the accelerometer can sense at least one ofacceleration change or angular acceleration change when the user movesthe remote controlling apparatus 200. The accelerometer can beimplemented using a three-axis accelerometer for measuring the increaseor decrease of the straight-line speed for three orthogonal axes.Information relating to the tilt can be obtained from the accelerometerusing motion acceleration related information of the remote controllingapparatus 200 and a gravity acceleration component of the static state.

The gyro sensor is an inertial sensor for measuring the angularrotational velocity of the remote controlling apparatus 200, that is, asensor for measuring the direction and the speed of the rotation usingthe inertial force of the rotating object. The gyro sensor can beimplemented using a three-axis angular velocity sensor for measuring theincrease or decrease of the rotation angle for three orthogonal axes.

The magnetometer is a sensor for measuring azimuth. That is, themagnetometer indicates the sensor for measuring the azimuth by detectingthe magnetic field generated around the earth south and north. Themagnetometer can be implemented using a three-axis magnetometer formeasuring intensity and direction of the magnetism for three orthogonalaxes. The north direction measured by the magnetometer can be themagnetic north. Even when the magnetometer measures the magnetic north,the true north may be output using internal calculations.

The detection unit 210 can further include a distance sensor optionallythe distance sensor measures the distance between the remote controllingapparatus 200 and the display apparatus 100. That is, the distancesensor can detect the distance between the user's usage location of theremote controlling apparatus 200 and the display apparatus 100.

The input unit 220 receives various user commands.

The input unit 220 may comprise a first button unit (not illustrated)which is used to receive a user command to select one of the pointingmode and the gesture mode in the operation mode of the display apparatus100, and a second button unit (not illustrated) which is used to receivea user command to select an object displayed on the UI screen.

If the first button unit (not illustrated) is pressed, informationregarding the operation mode may include information indicating that anexternal device is in the gesture mode, or if the pressed first buttonunit (not illustrated) is released, information regarding the operationmode may include information indicating that an external device is inthe pointing mode.

For example, the mode information is information indicating the pointingmode or the gesture mode, and may be implemented in diverse forms. Forexample, the mode information may be implemented in a flag form. In thecase of the pointing mode, the flag may indicate “0”, or in the case ofthe gesture mode, the flag may indicate “1”. That is, flag informationindicating “0” or “1” can be generated according to whether the firstbutton unit (not illustrated) is manipulated.

The signal output unit 340 outputs a remote controlling signal to thedisplay apparatus 100.

The signal output unit 340 may transmit the information regarding theremote controlling apparatus 200 to the display apparatus 100 usingvarious communication schemes such as BT, Zigbee, Wi-Fi, IR, SerialInterface, and USB.

The signal output unit 340 may output to the display apparatus 100 asignal corresponding to various user commands input through an inputunit 220.

The control unit 240 controls overall operations of the remotecontrolling apparatus 200. For example, the controller 240 may be aCentral Processing Unit (CPU) or a Microcontroller Unit (MCU).

The controller 240 may control the operations of the detection unit 210,the input unit 220, and the signal output unit 230.

If a user command is input through the above-described first button unit(not illustrated), the control unit 240 may control the signal outputunit 230 to output information regarding a user command to select one ofthe pointing mode and the gesture mode and information regarding thedetected motion.

The control unit may control the signal output unit 230 to outputinformation regarding a user command to select one of the pointing modeand the gesture mode consecutively while the first button unit (notillustrated) is pressed and not to output information regarding a usercommand to select one of the pointing mode and the gesture mode whilepressing on the first button unit (not illustrated) is released.

The external display apparatus 100 may maintain the pointing mode asdefault during a period where information regarding a user command toselect one of the pointing mode and the gesture mode is not receivedfrom the remote controlling apparatus 200.

The external display apparatus 100 may convert the operation mode to thegesture mode as a non-default mode and maintain the converted modeduring a period where information regarding a user command to select oneof the pointing mode and the gesture mode is received from the remotecontrolling apparatus 200 continuously.

An Exemplary Operation of the Remote Controlling Apparatus According tothe Second Exemplary Embodiment

Since the operation of the detection unit 210, the input unit 220, andthe signal output unit 230 in the second exemplary embodiment is similarto that in the first exemplary embodiment, detailed description thereofis not repeated.

Information regarding a user command to select one of the pointing modeand the gesture mode that is input through the input unit 220 may beused to generate a control signal inside the remote controllingapparatus 200 instead of being not transmitted to the display apparatus100. Accordingly, information regarding a user command to select one ofthe pointing mode and the gesture mode which is input through the inputunit 220, that is, the information regarding the operation mode may notbe implemented in the flag form unlike the first exemplary embodiment,but may be implemented in diverse forms capable of identifying input ofa user command to change the mode. The information regarding theoperation mode may be implemented in the flag form as in the firstexemplary embodiment.

The remote controlling apparatus 200 may generate a control signal tocontrol the display apparatus 100 using information regarding a usercommand to select one of the pointing mode and the gesture mode which isinput through the input unit 220 and motion information detected throughthe detection unit 210. FIG. 7 illustrates an exemplary configuration.

The control unit 240 may generate a control signal to control theexternal display apparatus 100 by analyzing corresponding informationthrough a predefined signal processing algorithm.

For example, the control unit 240 may control the signal output unit 230to analyze motion information through a gesture recognition algorithmbased on information regarding a user command to select a gesture modeand output a control signal corresponding to the analyzed gesture to thedisplay apparatus 100.

The control unit 240 may control the motion state of a pointing objectby analyzing motion information through a signal processing algorithmbased on information regarding a user command to select the pointingmode, calculating an absolute coordinates value to control the positionof the pointing object, and transmitting the calculated absolutecoordinates value to the display apparatus 100.

If a selection button unit 222 is selected, the control unit 240 maytransmit a corresponding signal to the display apparatus 100 so that anitem where the pointing object is located at the time when the selectionbutton unit 222 is selected can be selected in the display apparatus100. That is, in the second exemplary embodiment, the remote controllingapparatus 200 may generate a control signal to control the displayapparatus 100 by converting motion information detected through thedetection unit 210 to comply with the corresponding mode based oninformation regarding a user command to select one of the pointing modeand the gesture mode. Accordingly, the remote controlling apparatus 200operates in a pointing control mode or in a gesture control mode, unlikethe first exemplary embodiment.

FIG. 7 illustrates an exemplary remote controlling apparatus.

The detection unit 210 is a component for sensing the motion of theremote controlling apparatus 200. The detection unit 210 can includevarious sensors of an accelerometer 211, an angular velocity sensor 212,a magnetometer 213, and a touch sensor 214.

The accelerometer 211 measures the acceleration and the accelerationdirection when the motion occurs. The accelerometer 211 outputs thesensing value corresponding to the motion acceleration of the remotecontrolling apparatus 200 to which the sensor is attached, and thesensing value corresponding to the gravity acceleration varyingaccording to the tilt. Based on the output value of the accelerometer211, the control unit 260 can determine the tilt using the motionacceleration of the remote controlling apparatus 200 and the gravityacceleration component of the motionless state.

When the rotary motion occurs, the angular velocity sensor 212 detectsthe angular velocity by measuring Coriolis force acting in the velocitydirection. The control unit 260 can detect the rotation of the remotecontrolling apparatus 200 using the measurement value of the angularvelocity sensor 212.

The magnetometer 213 detects the magnetism of the earth or the magneticobject using a 2-axis or 3-axis fluxgate. The control unit 260 canmeasure the direction and the strength of the magnetism using themagnetic value detected by the magnetometer 213, and calculate theazimuth based on the measurement. The control unit 240 can determinewhich direction the remote controlling apparatus 200 rotates in.

The touch sensor 214 can detect the user's touch. The touch sensor 214can adopt a capacitive type or a resistive type. When a body part of theuser touches the surface of the remote controlling apparatus 200, thecapacitive touch sensor detects the electricity excited to the user bodyand calculates the touch coordinates using a dielectric coated on thesurface of the remote controlling apparatus 200. The resistive touchsensor includes two electrode plates embedded in the remote controllingapparatus 200. When the user touches, the resistive touch sensor detectsthe current flow of the upper and lower plates contacted at the touchedpoint and calculates the touch coordinates. Besides, IR detection type,surface acoustic wave type, integral stain gauge type, and piezoelectric type can be used to detect the touch, and their detaileddescriptions are omitted here.

The input unit 220 receives diverse user commands.

The input unit 220 may include a mode change button unit 221, aselection button unit 222, a 4-direction input unit 223, a touch inputunit 224, and a power button unit 225.

The mode change button unit 221 may receive a user command to select oneof the pointing mode and the gesture mode. The configuration of the modechange button unit 221 is disclosed.

The selection button unit 222 may receive a selection command.

The selection button unit 222 can receive a user command for selectingan item where the pointing object displayed on the screen of the displayapparatus 100 is located in the pointing mode. When the selection buttonunit 251 is pressed while the pointing object displayed on the screen ofthe display apparatus 100 is positioned on a particular item, thecorresponding object can be selected to execute the correspondingfunction. For example, when the corresponding content is an iconinterface of a specific application, the corresponding applicationexecution screen can be displayed.

The selection button unit 222 can function as an enter key or an OK keyaccording to characteristics of the provided UI screen.

The 4-direction input unit 223 may be disposed in an outer frame of theselection button unit 222, and receives the user command for 4-directionmanipulation.

The selection button unit 222 and the 4-direction input unit 223 caninclude at least one of a touch sensor and an Optical Joystick (OJ)adopting optical technology.

The touch input unit 224 can include a plurality of touch regions mappedto different functions. The touch input unit 224 can include theplurality of touch regions mapped to different functions such as channelchange function, volume control function, and menu function.

The power button unit 225 receives the user command for the powerON/OFF.

The remote controlling apparatus 200 may include a centering button unit(not illustrated) for mapping a reference state of the remotecontrolling apparatus 200 and the pointing object display location whenthe power ON is input to the power button unit 225. Alternatively, thevarious buttons as aforementioned may function as the centering buttonunit (not illustrated). For example, when the remote controllingapparatus 200 is turned on, the remote controlling apparatus 200 canpair with the display apparatus 100. When the pairing is completed andthe user presses the centering button unit (not illustrated), thepointing object can be displayed at the center of the display screen ofthe display apparatus 100 and the current state of the remotecontrolling apparatus 200 can be matched to the position of the pointingobject displayed at the center of the display screen. The referencepoint is at a center of the screen, by way of example, and the referencepoint may be set to another position on the display screen.

Operations of the control unit 240 can be executed by the program storedto the storage unit (250).

The storage unit (250) can store various data such as O/S software fordriving the remote controlling apparatus 200, and signal processingalgorithm (as in the second exemplary embodiment) for processing thedetection signal detected by the detection unit 210. Using the variousprograms stored to the storage unit 250), the control unit 240 controlsthe operations of the remote controlling apparatus 200.

The control unit 240 includes a RAM 241, a ROM 242, a main CPU 243,first through n-th interfaces 244-1 through 244-n, and a bus 245.

The RAM 241, the ROM 242, the main CPU 243, and the first through n-thinterfaces 244-1 through 244-n can be connected via the bus 245 totransmit and receive various data and signals.

The first through n-th interfaces 244-1 through 244-n may be connectedto, not only the components illustrated in FIG. 7, but also the othercomponents for access of the main CPU 243.

The main CPU 243 accesses the storage unit (250) and performs thebooting using the O/S stored to the storage unit (250). The main CPU 243performs various operations using the various programs and data storedto the storage unit (250).

The ROM 424 stores an instruction set for the system booting. When theturn-on command is input and the power is supplied, the main CPU 243copies the O/S stored to the storage unit (250), to the RAM 241according to the instruction stored to the ROM 242, and boots up thesystem by executing the O/S. When the booting is completed, the main CPU243 copies the various programs stored to the storage unit (250), to theRAM 241, and performs various operations by executing the program copiedto the RAM 241.

The control unit 240 can perform various operations by copying andexecuting the program stored to the storage unit (250), to the RAM 241.

FIG. 8 illustrates a signal processing algorithm according to anembodiment of the present invention.

As illustrated in FIG. 8, the signal processing algorithm includes afirst block 711 for receiving the sensor measurement value, a secondpreprocessing block 712, a third pose estimation block 713, a fourthmovement classification block 714, a fifth calibration block 715, asixth user intention analysis block 716, a seventh gain functionapplication block 717, and an eighth gesture recognition block 718. Thesignal processing algorithm of FIG. 7 can be performed by the CPU or theMCU.

The first block 711 receives the various sensor values from thedetection unit 220. For example, the first block 711 can receive thesensor value sensed by at least one of the accelerometer, the gyrosensor, and the magnetometer.

The second block 712 preprocesses the received sensor value. The secondblock 712 performs physical quantity conversion, sensor axistransformation, and low pass filtering for the sensor value.

For example, the sensor value converted to a digital value is convertedto the actual physical quantity (the physical quantity conversion)applicable to the signal processing algorithm. Axes of theaccelerometer, the angular velocity sensor, and the magnetometer can beset to one defined axis (the sensor axis transformation). The low passfiltering can remove electrical noise and unintended high frequency ofthe sensor (the low pass filtering).

The third block 713 estimates a pose or a pose angle (Euler angles(roll, pitch, yaw (heading)) from each sensor value. Estimation usingKalman filter can be applied.

The fourth block 714 classifies the movement state using the sensorsignal, and can determine whether the remote controlling apparatus 200does not move, moves slowly, or moves fast.

The fifth block 715 performs calibration. When the fourth block 714determines that the remote controlling apparatus 200 does not move, thatis, determine zero rate, the fifth block 715 can calculate an average ofthe output values of the angular velocity sensor, subtract the averagefrom the output value of the angular velocity sensor, and thuscompensate for an offset value of the angular velocity sensor.

The sixth block 716 analyzes and determines whether the user intends tomove, stop, or click the remote controlling apparatus 200.

The seventh block 717 converts the yaw angle and the pitch angle outputfrom the third block 713 to X and Y coordinates of the display apparatus100. Using the converted coordinates, the location of the mouse cursorcan be set.

The eighth block 718 may recognize the designated gesture using thesignal output form the second block 712.

The coordinates of the pointing object can be set in the UI screen ofthe display apparatus 100 by mapping the X and Y coordinates output fromthe seventh block 717 to the X and Y coordinates of the pointing object.

Using the Euler angles (roll, pitch, yaw) output from the eighth block718 or the X and Y coordinates output from the seventh block 717, apreset event can be generated in the UI screen to perform thecorresponding operation.

The calculations according to the signal processing algorithm can beprocessed by the control unit 260 of the remote controlling apparatus200 or the control unit 130 of the display apparatus 100.

FIGS. 9A, 9B, and 9C illustrate an exemplary exterior of a remotecontrolling apparatus 200. FIG. 9A is a perspective view of the remotecontrolling apparatus 200, FIG. 9B is a view taken in the direction b ofFIG. 9A, and FIG, 9C is a view taken in the direction c of FIG. 9A. FIG.10 illustrates of a plurality of mode change button units 231 of theremote controlling apparatus 200.

Referring to FIG. 9A, the upper front part of the remote controllingapparatus 200 may protrude more than the lower front part and the topside gradually inclines backwards with a small curvature by consideringthe pose of the hand naturally cupping so as to minimize fatigue of thehand in the usage.

The front side and the left/right sides of the remote controllingapparatus 200 may be rounded concavely to enhance grip as illustrated inFIG. 9C. Hence, the user can comfortably hold and use the remotecontrolling apparatus 200 with the hand without causing strain to thehand or the whist.

When the user holds the remote controlling apparatus 200 with the hand,the mode change button unit 231 can be disposed in the front side or thefront side and either side of the remote controlling apparatus 200 wherethe middle finger F2 and the ring finger F3 of the five fingers areplaced, and the selection button unit 251 can be disposed in the topside or the top front side of the remote controlling apparatus 200 wherethe thumb F1 is placed (FIG. 11).

The disposition of the mode change button unit 221 and the selectionbutton unit 222 can be set by considering the locations of the fingersF1 through F3 for naturally pressing the mode change button unit 221 andthe selection button unit 222 while holding the remote controllingapparatus 200 with the hand.

While the single mode change button unit 221 may be formed by way ofexample, a remote controlling apparatus 2000 can include a plurality ofmode change button units 2210 and 2211 as illustrated in FIG. 10. Themode change button units 2210 and 2211 may be placed in an up and downconfiguration so that the middle finger and the ring finger of the fivefingers can manipulate the mode change button units 2210 and 2211.

When a plurality of the mode change button units 2210 and 2211 isprovided, the mode change button units 2210 and 2211 may be manipulatedalone or together to generate different signals.

The 4-direction input unit 223 and the touch input unit 224 can beformed on the top side of the remote controlling apparatus 200. The4-direction input unit 223 can surround the selection button unit 222and the touch input unit 224 can surround the selection button unit 222.That is, the selection button unit 222 may be disposed in the innermostpart and the range expands outwards from the selection button unit 222to place the 4-direction input unit 223 and the touch input unit 224.

The disposition of the 4-direction input unit 223 and the touch inputunit 224 facilitates easy manipulation with the thumb which canrelatively freely move among the five fingers while gripping the remotecontrolling apparatus 200.

The touch input unit 224 can include a channel control region on theright, a volume control region on the left, and a plurality of touchregions mapped to different functions such as menu region on the upperside as illustrated in FIG. 9B.

The power button unit 225 can be disposed in the upper rear side of theremote controlling apparatus 200 to block the relatively easy access ofthe finger, compared to the button units as stated above. Thedisposition of the power button unit 225 can minimize the turning ofpower off if the power button unit 225 is pressed by mistake in use ofthe remote controlling apparatus 200.

While the usage of the remote controlling apparatus 200 is explainedbased on, but not limited to, the user who can use all of the fivefingers, the user can manipulate the button units of the remotecontrolling apparatus 200 with a most suitable finger or number offingers.

FIGS. 11A-11C illustrate exemplary manipulation of a remote controllingapparatus according to various embodiments of the present invention.

As illustrated in FIG. 11A, the user can select the pointing mode of thedisplay apparatus 100 by releasing the pressed mode change button unit221. As such, when the display apparatus 100 is in the pointing mode,the UI screen provided by the display apparatus 100 can display thepointing-type pointer.

The user can select the gesture mode of the display apparatus 100 bykeeping pressing the mode change button unit 221 as illustrated in FIG.11B. When the display apparatus 100 is in the gesture mode, the UIscreen provided by the display apparatus 100 can display the palm-typepointer.

The flicking and the zoom in/out can be controlled through the gestureof the remote controlling apparatus 200 in the corresponding UI screen.For example, the flicking for switching the displayed UI screen to thedifferent UI screen, the flicking for displaying a new menu screen, andthe zoom in/out for zooming in and out the displayed content arepossible.

The display apparatus 100 can operate basically in the pointing mode,and in the gesture mode if there is the grip which keeps pressing themode change button unit 221.

As illustrated in FIG. 11C, the user can control the panning by keeppressing the mode change button unit 221 and the selection button unit222. In this case, the UI screen provided from the display apparatus 100can display the grip-type pointer. For example, when the remotecontrolling apparatus 200 is rotated in a particular direction, thecontent image enlarged and displayed on the screen can be moved in thecorresponding direction. That is, when a particular content is selectedand the remote controlling apparatus 200 is rotated vertically andhorizontally, the displayed content image can move in the directioncorresponding to the rotation direction of the remote controllingapparatus 200. Hence, the grip-type pointer can provide the user withintuitive cognition like holding the content.

FIG. 12 illustrates operations in the pointing mode according to anembodiment of the present invention.

As illustrated in FIG. 12, when the display apparatus 100 is in thepointing mode, the display apparatus 100 can display the GUIcorresponding to the pointing mode. That is, the hand-shaped pointer 10of the pointing type can be displayed on the screen.

When the pointing-type pointer is put on a content A 121 and the userrotates the remote controlling apparatus 200 to the right, thepointing-type pointer 10 displayed on the screen can move to a content B122 on the right of the content A 121.

The pointing-type pointer 10 displayed on the screen can move accordingto the rotation direction and the rotation degree of the remotecontrolling apparatus 200 based on the absolute coordinate scheme. Theabsolute coordinate scheme moves the pointer in an absolute coordinateregion and the pointer displayed on the screen moves based on a presetreference value, which is different from the relative coordinate schemefor moving the pointer based on the previous pointer location.

FIG. 13 illustrates operations in the mode change according to anembodiment of the present invention.

When the display apparatus 100 is in the pointing mode as illustrated inthe first drawing of FIG. 13, the screen of the display apparatus 100can display the pointing-type pointer 10 indicating the pointing mode.

When the display apparatus 100 is switched to the gesture mode, thepointer 10 displayed on the screen of the display apparatus 100 ischanged to the palm-type pointer 11 indicating the gesture mode andguide GUIs 21-1 and 21-2 guiding the gesture input direction can bedisplayed. For example, when new system UI pages are arranged on theleft and the right as illustrated in the drawing, the guide GUIs 21-1and 21-2 of the left and right arrows indicating the left and rightflicking gestures can be displayed.

When the remote controlling apparatus 200 is rotated to the left, thepalm-type pointer 11 is changed to and displayed as a pointer 12 of thealtered angle in the same manner as the actual left flicking hand of theuser and the guide GUI 21-1 guiding the left direction can behighlighted and displayed. For example, when the flicking is appliedonto the touch screen, the hand angle of the user is generally changed.Similarly, the type of the pointer can be changed and displayed in thesame fashion. Thus, the intuitive cognition like flicking on the touchscreen can be provided to the user.

The movement direction of the UI screen displayed on the screen can becontrolled according to the rotation direction of the remote controllingapparatus 200. For example, when the remote controlling apparatus 200 isrotated to the left as illustrated in the drawing, the UI screen can bemoved to the left and displayed. That is, contents displayed in thecenter are moved to the left and the unseen contents on the right can bedisplayed.

FIG. 14 illustrates operations in the gesture mode according to anembodiment of the present invention.

As illustrated in FIG. 14, when a particular content is displayed on thescreen and the user presses the first button unit (e.g., the mode changebutton unit 221 of FIG. 4B) of the remote controlling apparatus 200, thedisplay apparatus 100 operates in the gesture mode and displays guideGUIs 22-1 through 22-4 guiding the available gesture inputs on thescreen.

When the user rotates the remote controlling apparatus 200 in aparticular direction while pressing the mode change button unit 221, apreset menu can be displayed in the region corresponding to the rotationdirection. For example, when the remote controlling apparatus 200 isrotated to the right as illustrated in the drawing, a content list canbe displayed on the left.

When the user releases the pressed mode change button, the displayapparatus 100 operates in the pointing mode. In this case, thepointing-type pointer 14 can move according to the rotation of theremote controlling apparatus 200. For example, the pointing-type pointer14 is positioned on a particular content C 123 and the user rotates theremote controlling apparatus 200 downwards, the pointing-type pointer 14can move to a content D 124 below according to the rotation degree.

When the pointing-type pointer 14 is positioned on the content D 124 andthe user presses the selection button of the remote controllingapparatus 200, the selected content D 124 is executed and displayed onthe screen.

FIG. 15 illustrates operations in the gesture mode according to anembodiment of the present invention.

As illustrated in FIG. 15, when a pointing-type pointer 15 is positionedon a content E 125 in the pointing mode and the user presses theselection button (e.g., the selection button unit 232 of FIG. 9B) of theremote controlling apparatus 200, the selected content E 125 isdisplayed on the screen of the display apparatus 100.

When the user presses the mode change button (e.g., the mode changebutton unit 221 of FIG. 9B) of the remote controlling apparatus 200, apointing-type pointer 15 displayed on the screen is changed to apalm-type pointer 16 indicating the gesture mode and guide GUIs 23-1 and23-2 guiding the available gesture inputs can be displayed.

When the user moves the remote controlling apparatus 200 away from theuser while pressing the mode change button, the content displayed on thescreen can be zoomed out and displayed. The movement away from the usercan rotate the remote controlling apparatus 200 toward the displayapparatus 100.

When the user moves the remote controlling apparatus 200 closer to theuser while pressing the mode change button, the content displayed on thescreen can be zoomed in and displayed. The movement closer to the usercan rotate the remote controlling apparatus 200 away from the displayapparatus 100.

FIG. 16 illustrates operations in the gesture mode according to anembodiment of the present invention.

As illustrated in FIG. 16, when the pointing-type pointer 15 is placedon the content E 125 in the pointing mode and the user presses thesecond button unit (e.g., the selection button unit 222 of FIG. 9B) ofthe remote controlling apparatus 200, the selected content E 125 isdisplayed on the screen of the display apparatus 100.

When the user keeps pressing the selection button while pressing themode change button (e.g., the mode change button unit 231 of FIG. 9B),the pointing-type pointer 15 is changed to a grip-type pointer 17indicating the panning manipulation and guide GUIs 24-1 through 24-4guiding the available gesture inputs can be displayed.

When the user rotates the remote controlling apparatus 200 verticallyand horizontally while pressing the mode change button and the selectionbutton, the content displayed on the screen can be panned. For example,when the user rotates the remote controlling apparatus 200 to the rightwhile pressing the mode change button and the selection button, thecontent displayed on the screen can be moved to the right.

While the user applies the panning manipulation while pressing the modechange button and the selection button at the same time in thisembodiment, the panning manipulation can be performed while pressing theselection button without pressing the mode change button.

FIG. 17 illustrates an exemplary control method of the remotecontrolling apparatus according to an embodiment of the presentinvention.

Referring to FIG. 17, the remote controlling apparatus 200 communicateswith the display apparatus 100 (S1610). For example, when the remotecontrolling apparatus 200 communicates with the display apparatus 100using the BT and the remote controlling apparatus 200 is turned on, theremote controlling apparatus 200 can discover and pair with the displayapparatus 100.

After the pairing, the display apparatus 100 can operate in the pointingmode. When the user command is input through the remote controllingapparatus 200, the pointing object can be displayed on the screen of thedisplay apparatus 100 and the displayed pointing object location can bemapped to the current state of the remote controlling apparatus 200.

The motion of the remote controlling apparatus 200 is detected (S1720).

If a user command to select one of the pointing mode and the gesturemode is input (S1730:Y), motion information detected in S1720 andinformation regarding a user command input in S1730 are output to thedisplay apparatus 100 (S1740).

The pointing mode may be a mode to control the motion of a pointingobject displayed on the UI screen of the display apparatus 100 accordingto the motion of the detected remote controlling apparatus, and thegesture mode may be a mode to control a display state of a UI screenaccording to the predefined gesture of the remote controlling apparatus200.

In the pointing mode, the movement state of the pointing objectdisplayed on the screen of the display apparatus 100 may be controlledaccording to the motion state of the remote controlling apparatus 200.The motion state can be the rotation state of the remote controllingapparatus 200.

In the gesture mode, the screen display state of the display apparatus100 may be controlled according to a movement of the remote controllingapparatus 200. For example, the screen can be changed to the menu screenor other UI screen.

When the user command not for the mode change is input, the remotecontrolling apparatus 200 can send the control signal corresponding tothe input command to the display apparatus 100. For example, when thecontent selection command or the pointing object movement command isinput, the remote controlling apparatus 200 can send the correspondingcontrol signal to the display apparatus 100.

FIG. 18 illustrates a control method of the display apparatus accordingto an embodiment of the present invention.

Referring to FIG. 18, the display apparatus 100 displays a UI screenhaving a pointing object for performing an indication function isdisplayed according to a preset event (S1810). The preset event may bean event in which the display apparatus 200 is connected to the remotecontrolling apparatus 200, or an event in which a predetermined buttonof the connected remote controlling apparatus 200 is pressed.

A remote control signal including motion information of the remotecontrolling apparatus and information regarding a user command to selectone of the pointing mode and the gesture mode is received from theremote controlling apparatus 200 (S1820).

One of the pointing mode and the gesture mode is selected according tothe information regarding a user command to select one of the pointingmode and the gesture mode which is received from the remote controllingapparatus 200 and the display apparatus 100 is operated accordingly(S1830).

The display apparatus 100 can display the hand-shaped pointing-typepointer on the UI screen and operates in the pointing mode whenreceiving information regarding a user command to select the pointingmode from the remote controlling apparatus 200, and display thehand-shaped pointer of the palm type and operates in the gesture modewhen receiving information regarding a user command to select thegesture mode from the remote controlling apparatus 200.

When the flicking manipulation is input through the remote controllingapparatus 200 in the gesture mode, the display apparatus 100 can displaythe hand-shaped pointer corresponding to the flicking hand gesture.

When the panning manipulation is possible in the gesture mode, thedisplay apparatus 100 can change and display the hand-shaped pointer tothe grip type. The hand-shaped pointers are disclosed herein.

The display apparatus 100 can display the guide GUI for guiding thegesture input in the gesture mode.

For example, when the flicking manipulation is possible, the guide GUIof the left and right arrows for guiding the left and right gestureinputs can be displayed. When the zoom in/out is possible, the guide GUIof the up and down arrows for guiding the up and down gesture inputs canbe displayed. When the panning manipulation is possible, the guide GUIof the up, down, left, and right arrows for guiding the up, down, left,and right gesture inputs can be displayed.

According to various embodiments of the present invention, the intuitiveremote control method can be provided to the user.

Methods according to various embodiments of the present invention can berealized by upgrading software of the existing display device or theuser terminal.

A non-transitory computer readable medium can store a program forsequentially carrying out the present control methods.

The non-transitory computer readable medium indicates a medium forstoring data semi-permanently and allowing reading of the device, ratherthan a medium for storing data for a short term such as register, cache,and memory. The various application and programs as stated above can bestored to and provided by the non-transitory computer readable mediumsuch as CD, DVD, hard disc, Blue Ray disc, USB, memory card, and ROM.

While the bus is depicted in the block diagrams of the display apparatusand the remote controlling apparatus, the communication between thecomponents of the display apparatus and the remote controlling apparatuscan be performed via the bus. Each apparatus may include a processorsuch as CPU and microprocessor for conducting the aforementionedoperations, for example.

Although a few embodiments of the present invention have beenillustrated and described, it would be appreciated by those skilled inthe art that changes may be made in these embodiments without departingfrom the principles and spirit of the invention, the scope of which isdefined in the claims and their equivalents.

What is claimed is:
 1. A remote controlling apparatus for providingremote control signals for an external display apparatus having aplurality of operation modes comprising a pointing mode and a gesturemode, comprising: a signal output unit for outputting the remote controlsignals to the external display apparatus which provides a userinterface screen; a detection unit for detecting motion of the remotecontrolling apparatus; an input unit comprising a first button unit anda second button unit, the first button unit being used for receiving auser command for selecting one of the pointing mode and the gesture modein the operation mode of the external display apparatus, and the secondbutton unit being used for receiving a user command for selecting anpointing object displayed in the user interface screen; and a controlunit for controlling the signal output unit to output informationregarding the detected motion and information regarding the user commandfor selecting one of the pointing mode and the gesture mode, when theuser command via the first button unit is input, wherein the pointingmode is a mode for controlling movement of the pointing object displayedon the user interface screen according to the motion of the remotecontrolling apparatus detected by the detection unit, and the gesturemode is a mode for controlling display state of the user interfacescreen according to a predefined gesture of the remote controllingapparatus.
 2. The remote controlling apparatus of claim 1, wherein,during a time interval in which the information regarding the usercommand for selecting one of the pointing mode and the gesture mode isnot received from the remote controlling apparatus, the external displayapparatus remains in the pointing mode as a default mode, and during atime interval in which the information regarding the user command forselecting one of the pointing mode and the gesture mode is continuouslyreceived from the remote controlling apparatus, the external displayapparatus transitions to and remains in the gesture mode as anon-default mode.
 3. The remote controlling apparatus of claim 2,wherein the control unit controls the signal output unit to output theinformation regarding the user command for selecting one of the pointingmode and the gesture mode continuously while the first button unit isbeing pressed, and not to output the information regarding the usercommand for selecting one of the pointing mode and the gesture mode whenthe pressed first button unit is released.
 4. The remote controllingapparatus of claim 1, wherein the detection unit comprises at least oneof an accelerometer, an angular velocity sensor, and a magnetometer. 5.A display apparatus having a plurality of operation modes comprising apointing mode and a gesture mode, controlled by a remote controllingapparatus which provides remote control signals, comprising: a receivingunit for receiving the remote control signals comprising informationregarding a motion of the remote controlling apparatus, informationregarding a user command for selecting one of the pointing mode and thegesture mode, and information regarding an item selection from theremote controlling apparatus; a display unit for displaying a userinterface screen having a pointing object which performs an indicatingfunction on the user interface screen; and a control unit forcontrolling to select one of the pointing mode and the gesture modebased on the information regarding the user command for selecting one ofthe pointing mode and the gesture mode received from the remotecontrolling apparatus, wherein the pointing mode is a mode forcontrolling movement of the pointing object displayed on the userinterface screen according to the motion of the remote controllingapparatus detected by the detection unit, and the gesture mode is a modefor controlling display state of the user interface screen according toa predefined gesture of the remote controlling apparatus.
 6. The displayapparatus of claim 5, wherein the display unit displays the pointingobject of which a shape is changed according to the operation mode. 7.The display apparatus of claim 5, wherein the control unit determinesthe operation mode of the display apparatus as the pointing mode whilethe receiving unit does not receive the information regarding the usercommand for selecting one of the pointing mode and the gesture mode, andthe control unit determines the operation mode of the display apparatusas the gesture mode while the receiving unit receives the informationregarding the user command for selecting one of the pointing mode andthe gesture mode.
 8. The display apparatus of claim 5, wherein, when theoperation mode of the display apparatus is the pointing mode, a menuitem included the user interface screen is selected based on theinformation regarding a motion of the remote controlling apparatus andthe information regarding an item selection received from the remotecontrolling apparatus.
 9. The display apparatus of claim 6, wherein thecontrol unit controls to display the pointing object by changing atleast one of a shape, a color, a size, a location, and a directionaccording to an input manipulation state on the user interface screen.10. The display apparatus of claim 6, wherein the pointing object is ahand-shaped pointer, and the control unit controls to display thehand-shaped pointer by changing into a pointing type, a palm type, or agrip type based on a possible manipulation of flicking, zoom in/out, andpanning, on the user interface screen.
 11. A control method of a remotecontrolling apparatus for providing remote control signals for anexternal display apparatus having a plurality operation modes comprisinga pointing mode and a gesture mode, comprising: detecting motion of theremote controlling apparatus; outputting information regarding thedetected motion and information regarding a user command for selectingone of the pointing mode and the gesture mode, when the user command forselecting one of the pointing mode and the gesture mode in the operationmode of the external display apparatus is input, wherein the pointingmode is a mode for controlling movement of a pointing object displayedon the user interface screen according to the motion of the remotecontrolling apparatus, and the gesture mode is a mode for controllingdisplay state of the user interface screen according to a predefinedgesture of the remote controlling apparatus.
 12. A control method of adisplay apparatus having a plurality of operation modes comprising apointing mode and a gesture mode, controlled by a remote controllingapparatus which provides remote control signals, comprising: displayinga user interface screen having a pointing object for performing anindicating function; receiving the remote control signals comprisinginformation regarding a motion of the remote controlling apparatus andinformation regarding a user command for selecting one of the pointingmode and the gesture mode from the remote controlling apparatus; andselecting one of the pointing mode and the gesture mode based on theinformation regarding the user command for selecting one of the pointingmode and the gesture mode received from the remote controllingapparatus.